The relationship between classical item characteristics and item response time on computer-based testing / 한국의학교육

PURPOSE: This study investigated the relationship between the item response time (iRT) and classic item analysis indicators obtained from computer-based test (CBT) results and deduce students' problem-solving behavior using the relationship. METHODS: We retrospectively analyzed the results of the Comprehensive Basic Medical Sciences Examination conducted for 5 years by a CBT system in Dankook University College of Medicine. iRT is defined as the time spent to answer the question. The discrimination index and the difficulty level were used to analyze the items using classical test theory (CTT). The relationship of iRT and the CTT were investigated using a correlation analysis. An analysis of variance was performed to identify the difference between iRT and difficulty level. A regression analysis was conducted to examine the effect of the difficulty index and discrimination index on iRT. RESULTS: iRT increases with increasing difficulty index, and iRT tends to decrease with increasing discrimination index. The students' effort is increased when they solve difficult items but reduced when they are confronted with items with a high discrimination. The students' test effort represented by iRT was properly maintained when the items have a 'desirable' difficulty and a 'good' discrimination. CONCLUSION: The results of our study show that an adequate degree of item difficulty and discrimination is required to increase students' motivation. It might be inferred that with the combination of CTT and iRT, we can gain insights about the quality of the examination and test behaviors of the students, which can provide us with more powerful tools to improve them.

Evidências de validade da Bateria Informatizada de Linguagem Oral com prova de raciocínio / Evidence of validity of Computerized Oral Language Battery with reasoning assessment

Este estudo buscou relacionar a compreensão da linguagem oral com a habilidade de raciocinar, visando evidenciar validade correlacional para a Bateria Informatizada de Linguagem Oral por meio da associação com a Escala de Maturidade Mental Colúmbia. Participaram do estudo 82 alunos com idade entre 5 e 9 anos (M=7,05; DP=1,25), sendo 47,6% do sexo feminino. Frequentavam o pré (31,7%) e a 1ª (32,9%) e 2ª (35,4%) séries do Ensino Fundamental de escola pública do interior paulista. Aplicou-se coletivamente a Bateria e individualmente a Escala de Maturidade Mental Colúmbia. A Análise de Variância, por série, indicou diferença significativa para as provas de Compreensão Morfossintática, Sequência Lógica, Organização Lógico-Verbal e escore total da Bateria. A análise de regressão revelou que o raciocínio geral representa 29% das habilidades necessárias para um bom desempenho na Bateria. Verificou-se correlação baixa entre escores totais da Bateria e do Colúmbia, porém muito significativa. Constataram-se evidências de validade correlacional e de critério, para a Bateria.

The aim of this study was to relate the understanding of oral language to the ability to reason, in order to seek correlational validity evidence of the Computerized Oral Language Battery (Bateria Informatizada de Linguagem Oral) by association with the Columbia Mental Maturity Scale. The participants were 82 schoolchildren between 5 and 9 years of age (M=7.05, SD=1.25). They attended pre-school (31.7%),1st (32.9%) and 2nd (35.4%) grades of elementary public school in São Paulo and 47.6% of whom were girls. The Bateria Informatizada de Linguagem Oral was applied collectively in the school informatics laboratory and the Columbia Mental Maturity Scale, individually. Analysis of Variance for the series, indicated significant statistics differences for the scores of Morphosyntactic prove, Logical sequence prove, Logical Organization prove and for the total score of Bateria Informatizada de Linguagem Oral. Regression analysis revealed that general reason represents 29% of the skills needed to perform well in Bateria Informatizada de Linguagem Oral. There was low, but very significant correlation between total scores of Bateria Informatizada de Linguagem Oral and Columbia Mental Maturity Scale. These results indicated validity evidences to Bateria Informatizada de Linguagem Oral.

Experiences with Establishing and Implementing Learning Management System and Computer-Based Test System in Medical College / 한국의학교육

PURPOSE: The purpose of this study was to establish an e-learning system to support learning in medical education and identify solutions for improving the system. METHODS: A learning management system (LMS) and computer-based test (CBT) system were established to support e-learning for medical students. A survey of 219 first- and second-grade medical students was administered. The questionnaire included 9 forced choice questions about the usability of system and 2 open-ended questions about necessary improvements to the system. RESULTS: The LMS consisted of a class management, class evaluation, and class attendance system. CBT consisted of a test management, item bank, and authoring tool system. The results of the survey showed a high level of satisfaction in all system usability items except for stability. Further, the advantages of the e-learning system were ensuring information accessibility, providing constant feedback, and designing an intuitive interface. Necessary improvements to the system were stability, user control, readability, and diverse device usage. CONCLUSION: Based on the findings, suggestions for developing an e-learning system to improve usability by medical students and support learning effectively are recommended.

Emerging tasks of specialty certifying examination: educational measurement considerations

Medical specialty systems were launched in 1951 by the National Medical Services Law. The following year, the specialty certifying examination had implemented in the form of portfolio evaluation. A paper-and-pencil type examination was implemented in 1960, and the 55th examination was carried out in January 2012. Currently, 26 specialties are represented, and the overall pass rate is over 90%. The examination consists of a step 1 paper-and-pencil test and step 2 skills test. In the step 1 test, the test items are multiple choice questions and short answer questions. Clinical performance examination is partially applied to the step 2 test. To cope with changes in the social situation and the growth of medical services, developmental changes are needed in the specialty certifying examinations. Performance assessment is an alternative worth considering. CPX should be a major part of the skill test. A computer-based test should be introduced as soon as possible, and it could eventually be developed into an adaptive test.

Ontological analysis of a Computer Based Assessment Developer fit for the School of Medicine.

We designed and implemented a computer based assessment developer tool fit for the school of medicine. Issues about the implementation include the ease of usage, being user friendly and convenient conversion of PBT (Paper Based Test- existing exam items) material to CBT (Computer Based Test). Assessors with a little drag and drop can create exam papers in no more than a minute. The unique format of assessment upload, with related conversion and some mistake correction covered by automation of ontological relations are special advantages of this implementation. What makes this architecture well designed for the school of medicine is that the software techniques being used to handle upload and download of large size files (e.g. medical images). A large population can be assessed using this method without hindering the service.

Development of Computer-Based Test (CBT) and Student Recognition Survey on CBT / 한국의학교육

PURPOSE: The purpose of this study is to understand student recognition of CBT as well as its strengths and weaknesses, and to explore the improvement methodologies for the effective development and implementation of CBT. METHODS: A questionnaire survey was conducted twice (before and after implementation of CBT) with a total of 17 multiple-choice and 2 essay-type questions. The multiple-choice questions were analyzed by frequency analysis and the essay-type questions were coded by content analysis. RESULTS: The results are as follows. First, the overall satisfaction with CBT was shown to be high. Second, students listed the merits of CBT as follows: simple correction of answers (before: 89.4%; after: 80.8%), presentation of realistic materials (before: 72.9%; after: 84.7%), prompt feedback on grades (before: 60.3%; after: 71.1%), shortened exam time and effective time scheduling (before: 86.5%; after: 66.4%), accurate estimation of abilities (before: 70.2%; after: 36.6%), and assistance in academic improvement (before: 70.9%; after: 22.1%). Drawbacks of CBT were: inconvenience of review (before: 70.9%; after: 22.1%), inconvenient screen organizations (before: 0%; after: 48.1%), possibility of cheating (before: 73.9%; after: 31.8%), and equality issue of test (before: 47.3%; after: 17.3%). CONCLUSION: Assessment paradigms are currently shifting from summative evaluation to formative evaluation, from one-off assessment to continuous assessment, and from output assessment to process assessment. Therefore, CBT must be expanded to move from result-oriented summative evaluation to formative evaluation continuously monitoring the student learning process.